Dynamic Resource Provisioning in Grid Environments

Batch schedulers commonly used to manage access to parallel computing clusters are not typically configured to enable easy configuration of application-specific scheduling policies. In addition, their sophisticated scheduling algorithms can be relatively expensive to execute. Thus, for example, applications that require the rapid execution of many small tasks often do not perform well. Frey proposed that these problems be overcome by separating the two tasks of provisioning and scheduling. The provisioning component uses batch submissions (Condor “schedd” services in Frey’s work) to acquire resources for application execution. The scheduling component dispatches application tasks to those resources. We introduce here a dynamic resource provisioning Web Services-based architecture, DRP, and use this architecture to evaluate new methods designed to optimize both the dynamic resource provisioning and task scheduling within dynamically provisioning resource sets. The task scheduling is implemented in a separate system DeeF, a distributed execution environment framework, which has DRP integrated to offer the necessary dynamic resource provisioning for generic execution of arbitrary codes on the DRP managed resources. Based on our performance evaluation, DRP can allocate resources in less than a minute after which a pool of resources can be maintained, increased, and decreased based on the load of the application using DRP. Our execution framework DeeF can process 100K fine granular tasks in 160 seconds; in such a high throughput workload, the overhead per task is 1.6 ms per task, a low enough cost that DeeF and DRP can enable a wide range of applications to run significantly more efficiently, due to efficient task dispatch and the dynamic resource provisioning that makes the compute resource management trivial from the application’s viewpoint. We also address the performance implications various security mechanisms have on the execution framework and the dynamic resource provisioning.